Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enable tracing behind feature flag #159

Merged
merged 6 commits into from
Sep 12, 2023
Merged

Conversation

crawford
Copy link
Contributor

@crawford crawford commented Sep 8, 2023

This adds the "tracing" feature, which enables tokio's tracing
functionality. This allows tokio-console to attach to a running
instance and display interesting (and hopefully helpful) information
about the async tasks. In addition to the feature, the
"tokio_unstable" config option needs to be passed to rustc:

RUSTFLAGS="--cfg tokio_unstable" cargo build ...

In order to allow multiple layers of the enclave to be measured, the
console subscriber in each binary does not run on the default
port. You'll need to specify which one you want to connect to:

  • 51000 - odyn
  • 51001 - enclaver-run (supervisor)
  • 51002 - enclaver

Note that both odyn and enclaver-run are configured by default to
listen on all interfaces. This is to make it easier to connect
tokio-console to these processes which run in a seperate network
context, but be careful if running manually or with --net=host, as
anyone on the network will be able to connect.

Also note that when enclaver creates the container image tarball (via
tokio_tar), many tasks are spawned; so many that it will likely
exhaust the memory of your machine. Consider lowering the retention
time substantially using TOKIO_CONSOLE_RETENTION. 1s seemed to
work okay for a machine with 4 GiB free.

This doesn't change the way it's interpreted, but it was bugging me.
This makes it tricky to add optional development features, since we
might not want to release with those features enabled.
@crawford crawford force-pushed the tracing branch 4 times, most recently from c00cd4e to 689401e Compare September 8, 2023 19:07
The build failures can be rather confusing for folks who aren't using
Docker Desktop. At least with buildx in the invocation, there's a hint
that an additional component is necessary.
This allows you to use `cargo run` without having to specify which
binary.
This adds the "tracing" feature, which enables tokio's tracing
functionality. This allows `tokio-console` to attach to a running
instance and display interesting (and hopefully helpful) information
about the async tasks. In addition to the feature, the
"tokio_unstable" config option needs to be passed to rustc:

    RUSTFLAGS="--cfg tokio_unstable" cargo build ...

In order to allow multiple layers of the enclave to be measured, the
console subscriber in each binary does not run on the default
port. You'll need to specify which one you want to connect to:

- 51000 - odyn
- 51001 - enclaver-run (supervisor)
- 51002 - enclaver

Note that both odyn and enclaver-run are configured by default to
listen on all interfaces. This is to make it easier to connect
tokio-console to these processes which run in a seperate network
context, but be careful if running manually or with `--net=host`, as
anyone on the network will be able to connect.

Also note that when enclaver creates the container image tarball (via
tokio_tar), many tasks are spawned; so many that it will likely
exhaust the memory of your machine. Consider lowering the retention
time substantially using `TOKIO_CONSOLE_RETENTION`. `1s` seemed to
work okay for a machine with 4 GiB free.
@crawford
Copy link
Contributor Author

This turned into a bit of a catch-all. Let me know if you want this split into separate PRs.

Copy link
Contributor

@eyakubovich eyakubovich left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM -- I'm sure this will be useful next time we need to debug!

@crawford crawford merged commit 30395e5 into edgebitio:main Sep 12, 2023
3 checks passed
@crawford crawford deleted the tracing branch September 12, 2023 15:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants